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1.  INTRODUCTION 


1 .1 .  Executive  Summary 

This  report  summarizes  the  requirements,  methods,  procedures,  and  accomplishments 
of  the  National  Air  Intelligence  Center  (NAIC)  Software  Process  Improvements  effort. 
Harris  Technical  Services  Corporation  (HTSC)  performed  the  work  as  a  subcontractor 
to  Harris  Corporation  under  Rome  Laboratory  contract  F30602-94-D-0055,  TOA  Task 
#5.  This  report  satisfies  data  item  A005  of  this  task. 


1.2.  NAIC  Overview 

NAIC  acquires,  collects,  analyzes,  produces,  and  disseminates  foreign  aerospace 
intelligence.  It  conducts  integrated  analysis  programs  to  meet  the  intelligence 
production  requirements  of  the  Air  Intelligence  Agency;  the  Air  Force  Assistant  Chief  of 
Staff,  Intelligence;  and  the  Defense  Intelligence  Agency.  It  provides  finished  aerospace 
intelligence  to  the  operational  commands  of  the  USAF  and  JCS  and  serves  as  the 
single  Air  Force  intelligence  production  center.  To  meet  its  intelligence  missiori,  NAIC 
operates  an  intelligence  data  handling  system  and  collaborates  with  other  organizations 
to  improve  the  collection,  acquisition,  and  use  of  aerospace  intelligence.  In  addition,  the 
NAIC  assesses  the  capabilities  and  intentions  of  foreign  aerospace  forces,  weapon 
systems,  and  technologies  -  both  current  and  future.  NAIC  determines  the  implications 
of  these  capabilities  and  the  intentions  of  operational,  acquisition,  and  policy-making 
customers. 

To  accomplish  this  mission  requires  significant  automation  capabilities.  NAIC/SC  is  the 
focal  point  for  that  information  technology.  NAIC/SC  designs,  implements,  operates,  and 
maintains  the  technical  production  infrastructure  necessary  to  support  the  NAIC 
mission.  This  includes  providing  technical  computing  services,  internal  and  external 
communications,  and  research  and  development  of  production  systems.  NAIC/SC 
provides  production  support  for  NAIC  intelligence  products  and  computing  needs.  It 
does  so  by  acquiring,  developing,  implementing,  and  maintaining  application  software 
and  technologies  necessary  to  support  its  analytical  processes  and  automated 
information  handling  requirements. 

NAIC/SC  is  currently  migrating  its  existing  legacy  systems  to  provide  near  term 
enhanced  information  technology  capabilities  to  its  customers  and,  based  upon  a 
detailed  strategic  plan,  a  long  term  goal  of  providing  an  environment  to  increase 
customer  productivity  and  customer  satisfaction.  This  migration  of  legacy  systems  is 
being  accomplished  by  reengineering  existing  systems  into  an  open  systems 
architecture  using  relational  database  technology,  consistent  user  interfaces,  standard 
development  architecture,  and  standard  hardware  platforms.  To  be  able  to  meet  NAIC 
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software  migration  goals,  NAIC/SC  has  undertaken  an  extensive  effort  to  improve  its 
software  development  capabilities. 

NAIC/SCD  is  a  small  software  support  division,  50  people  total  supporting  the 
intelligence  community.  6  people  manage  problem  domains  and  projects,  36  people 
develop  software  and  data  base  systems  and  8  people  provide  software  support. 
Projects  are  small  and  developed  quickly.  The  small  projects  and  limited  project  staff 
size  bring  special  problems  in  tailoring  the  Software  Engineering  Institute’s  (SEI) 
Capability  Maturity  Model  (CMM).  Some  activities  that  are  needed  and  necessary  for 
large  development  and  maintenance  organizations  are  overkill  on  small  projects. 


1 .3.  Objectives  of  the  Effort 

Provide  engineering  and  software  process  improvement  support  for  NAIC/SCD  to  reach 
SEI’s  CMM  level  2  in  process  maturity.  While  working  to  reach  CMM  level  2,  the  goal 
was  to  develop  policies  and  practices  that  would  lead  directly  to  and  support  CMM  Level 
3.  Work  included  estimation,  software  subcontract  management,  software  configuration 
management,  software  quality  assurance,  metrics  and  measurements,  and  training. 

Dr  Phil  Koltun,  Harris  Engineering  Productivity  Group,  conducted  a  self-assessment 
workshop  from  3  through  6  Sept  96.  Following  the  self-assessment  workshop,  efforts 
focused  on  correcting  weaknesses  identified  at  the  September  workshop.  The  plans 
called  for  a  Software  Engineering  Institute's  CMM  Based  assessment  for  Internal 
Process  Improvement  (CBA-IPI)  in  October  of  1997. 


1.4.  Report  Overview 

The  remainder  of  this  report  is  as  follows; 

•  Section  2,  Contractual  Requirements,  summarizes  the  program  objectives  and  SOW 
requirements. 

•  Section  3,  Approach,  Methods,  &  Results,  describes  the  approach  taken  and  any 
special  methods  use  and  the  results. 

•  Section  4,  Recommendations,  document  HTSC  recommendations  to  NAIC/SCD. 

•  Section  5,  Lessons  Learned,  documents  a  few  lessons  from  the  NAIC  SPI  effort. 


•  Appendix  A,  Contract  Deliverables,  list  the  documents  developed  by  HTSC  in 
chronological  or  deliver  order. 
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•  Appendix  B,  Workshop  Findings,  summarizes  the  findings  from  the  workshop 
conducted  by  Dr,  Phil  Koltun. 

•  Appendix  C,  Summary  of  "A  Guide  to  Achieving  SEI  CMM  Level  2  at  NAIC" 

•  Appendix  D,  Acronym  Listing 
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2.  Contractual  Requirements 


2.1.  Summary 

This  section  contains  a  summary  of  the  program  objectives  and  SOW  requirements. 
The  SOW  required  HTSC  to  provide  software  process  improvement  (SPI)  support  plus 
system  and  software  process  training.  The  first  priority  of  HTSC  was  to  concentrate  on 
assisting  NAIC/SCD  in  establishing  sound  CMM  level  2  practices  as  they  applied  to 
NAIC  and  the  CMM.  This  support  manifested  itself  as: 

•  Reviewing  and  commenting  on  current  NAiC/SCD  software  development  practices 

•  Researching  industry  solutions  to  SPI  related  problems  relevant  to  NAIC/SCD 

•  Assisting  in  the  design,  impiementation,  and  training  of  process  improvements 

Upon  completion  of  analyzing,  designing,  implementing,  and  training  CMM  level  2 
processes,  HTSC  focused  on  the  CMM  level  3  practices  to  the  extent  allowed  by  the 
effort  and  as  directed  by  the  Software  Process  improvement  Steering  Group  (SPISG) 
and  the  Software  Engineering  Process  Group  SEPG. 

This  effort  was  be  documented  in  monthly  program  progress  reports  (A001)  and  other 
specified  reports.  The  specified  reports  include: 

•  An  assessment  plan  and  follow-up  recommendations  report  (CDRL  A002) 

•  Presentation  material  to  educate  NAIC/SCD  personnel  on  the  benefits  of  the  SPI 
effort  (CDRL  A003,  CDRL  A006,  CDRL  A008) 

•  A  guide  to  achieving  SEI  CMM  level  2  at  NAIC  (CDRL  A004) 

•  A  final  scientific  and  Technical  report  (CDRL  A005) 

•  A  NAIC/SCD  PnP  Configuration  Management  Chapter  (CDRL  A007) 

The  remaining  parts  of  this  section  document  the  individual  requirements  of  these 
reports. 


2.2.  Monthly  Program  Progress  Reports 

HTSC  ensured  progress  was  consistent  with  the  requirements  of  the  SEI  CMM  Level  2 
and  consistent  with  future  goals  of  attaining  SEI  CMM  Level  3.  HTSC  provided  a 
monthly  report  with  status  of  the  effort  and  reported  progress  toward  accomplishment  of 
contract  requirements. 
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2.3.  Assessment 


HTSC  assessed  on  going  NAIC/SCD  work  and  improvement  efforts  to  detem  ne 

requirements  for  reaching  Level  2  of  the  Software  Engineering 

Capability  Maturity  Model  (CMM).  The  purpose  of  this  assessment  was  to  determine  the 

maturity  of  the  unit’s  software  processes. 


2.3.1.  Assessment  Plan 

HTSC  created  an  Assessment  Plan  that  would  provide  NAIC/SC  with  guidance  for 
performing  SEI  CMM  self-assessment.  The  plan  contains  sufficient  detail  so  that 
government  personnel  can  repeat  the  assessment  process  on  future  projects.  Topi 
covered  include: 


•  On-going  Software  development  and  maintenance  projects 

.  Determining  the  current  level  of  NAIC/SC  plans  and  policies  lor  process 
improvement. 


2.3.2.  Conduct  a  CMM  assessment 

HTSC  conducted  a  SEI  CMM  assessment  of  projects  designated  by  the  Software 
Engineering  Process  Group  (SEPG).  Two  Government  representatives  in 

the  assessment  to  learn  the  proper  procedures  and  methods  used  in  conducting  an 

assessment. 


2.3.3.  Document  Assessment  Results 

Upon  completion  of  the  assessment,  the  assessment  team  documented  the  following 
areas; 


•  Strengths 

•  Weakness 

•  Inconsistencies,  gaps  and  duplication  of  efforts 

•  Improvement  suggestions 


Additionally,  HTSC  provided  recommendations  to  NAIC/SCD  for  meeting  the  SEI  CMM 
Level  2  requirements.  These  recommendations  should  be  consistent  with  a  long-term 
goal  of  attaining  SEI  CMM  Level  3. 


2.4.  Presentation  material 
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HTSC  provided  presentation  material  for  NAIC/SCD  to  educate  the  Center’s  personnel 
on  the  benefits  of  the  SEI  efforts.  The  goal  of  this  familiarization  was  to  create 
awareness,  within  NAIC,  of  the  commitment  and  resources  required  to  effect  software 
process  improvement  in  the  following  critical  areas: 

•  Developing  and  documenting  software  policy  and  procedures. 

•  Managing  software  activity  requirements. 

•  Developing  and  executing  a  software  development  plan  for  future  projects. 

•  Developing  and  executing  configuration  management  and  software  quality 
assurance  plans 

•  Developing  and  executing  a  commitment  process. 

HTSC  developed  and  delivered  process  training  on  NAIC/SCD's  standard  lifecycle 
models.  This  training  included  high-level  overview  training  and  detail  level  process 
training.  The  detail  level  training  modules  are  intended  for  "just-in-time"  training  at  the 
beginning  of  each  lifecycle  phase.  The  deliverable  was  in  the  form  of  training 
presentation  sides,  training  aids,  and  instructor’s  notes. 


2.5.  A  guide  to  achieving  SEi  CMM  ievei  2  at  NAiC 

HTSC  developed  a  guide  to  achieve  the  SEI  CMM  Level  2  goal.  For  each  of  the 
following  policies  and  processes:  recommend  areas  for  improvement  that  will  allow 
NAIC/SCD  to  move  toward  SEI  CMM  Level  2  compliance:  and  recommend  areas  that 
can  be  automated. 

The  areas  they  reviewed  were: 

•  Contract  Procurement  and  Management 

•  Project  Planning  and  Tracking 

•  Change  and  Status  Accounting 

•  Procedures  for  coordination  and  guidance  of  the  Software  Development  and 
Maintenance  Process 


2.6.  A  final  scientific  and  technicai  report 

HTSC  documented  all  technical  work  accomplished  and  information  gained  during 
performance  of  this  effort.  This  document  includes  all  pertinent  observations,  nature  of 
problems,  positive  and  negative  results,  and  design  criteria  established  where 
applicable.  It  documents  procedures  followed,  processes  developed,  “Lessons 
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Learned”,  etc.  It  documents  the  details  of  all  technical  work  to  permit  full  understanding 
of  the  techniques  and  procedures  used  in  evolving  technology  or  processes  developed. 
It  also  cross-references  separate  design,  engineering,  and  process  specifications  which 
were  delivered  in  order  to  permit  a  full  understanding  of  the  total  acquisition. 


2.7.  A  NAIC/SCD  PnP  Configuration  Management  Chapter 

HTSC  developed  a  NAIC/SCD  PnP  chapter  on  configuration  managemerit.  This  chapter 
includes  policy  and  processes  to  inject  configuration  management  activities  into  current 
NAIC/SCD  standard  lifecycle  models. 
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3.  Approach,  Method,  &  Results 

This  section  describes  the  approach  taken,  any  special  methods,  and  results. 

The  overall  method  to  process  improvement  was  to  investigate  and  document  current 
procedures.  Then  study  current  procedures  and  look  for  improvements.  Where 
procedures  did  not  exist,  SCO’s  SEPG  would  create  a  team  to  collect  information  then 
suggest  and  develop  a  new  procedure. 


3.1.  Orientation  &  Background  information 

Mr.  Don  Blankenship,  HTSC,  started  the  NAIC  SPI  effort  on  15  April  1996.  The 
Government  Furnished  Equipment  and  office  in  the  NAIC  holding  area  were  adequate 
to  begin  the  task. 

Capt.  Fulton  provided  newcomers  orientation  and  brief  history  of  NAIC  SPI  efforts.  Mr. 
Blankenship  met  many  members  of  NAIC/SCD  and  studied  the  provided  NAIC  Policy 
and  Procedures  manual  and  other  process  related  information. 


3.2.  Contract  Issues  &  Plan  to  meet  Deliverables  (CDRL  List) 

While  studying  the  process  artifacts,  Mr.  Blankenship  studied  the  contract  and  built  a 
work  breakdown  structure  (WBS).  From  the  WBS,  Mr.  Blankenship  scheduled  the 
contract  deliverables  (CDRL). 

Maj.  Dan  Burke  (SCDQ),  Capt.  Greg  Fulton  (SCDQ),  Ms  Tandi  Paugh  (Rome  Lab),  and 
Mr.  Blankenship  (HDSC)  met  on  3  May  96  to  discuss  contract  issues.  Draft  copies  of 
the  TIM  slides  were  reviewed  and  discussed.  Issues  and  concerns  were  identified  and 
solutions  agreed  to  by  all  parties. 

HTSC  conducted  the  first  Technical  Interchange  Meeting  (TIM)  on  7  May  96.  Mr.  Roger 
Beauman  started  the  presentation  with  information  on  software  process  improvement. 
He  gave  reasons  for  doing  process  improvement  and  results  from  Harris’s  software 
process  improvement  effort.  CDRL  milestones  were  presented  and  the  customer 
agreed  to  the  plan  of  attack  and  expected  results.  Customer  requested  the  next  TIM  be 
held  on  10  July  1996.  HTSC’s  goal  was  "Total  Customer  Satisfaction".  HTSC  planned 
to  meet  the  goal  with  100%  on-time  delivery  of  CDRL  products  and  delivery  of  value- 
added  support  to  NAIC/SCD’s  software  process  improvement  effort. 

CDRL  delivery  status,  milestones  and  short  term  goals  were  presented  at  the  10  July 
1996  Technical  Interchange  Meeting.  Col.  H.  Wayne  Wolfe  (SC),  Maj.  Dan  Burke 
(SCDQ),  and  Capt.  Greg  Fulton  (SCDQ)  attended  for  the  government.  Mr.  Don 
Blankenship,  Mr.  Roger  Beauman  and  Ms.  Debbie  Crow  from  Harris  attended. 
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ni<5rii‘5sions  were  had  that  more  concentration  needed  to  be  focused  on  the 
Configuration  Management  Working  Group  rather  than  the  Estimation  Working  Group. 

HTSC  presented  CDRL  delivery  status,  milestones  and  jj®  ne 

October  1996  Technical  Interchange  Meeting.  Attendees  included. 

Wolfe  and  Capt.  Greg  Fulton  from  NAIC/SC,  Ms.  Tandi  Paugh  from  Rorne  Lab.  and  Mr^ 

Roger  Beauman  from  Harris.  Ms  Paugh  provided  some  04?  These 

AcSeving  SEI  CMM  Level  2  at  NAIC”  technical  information  report  (CDRL  A004).  These 

changes  were  made  before  the  report  was  delivered. 

CDRi  deliverv  status,  milestones  and  short  term  goals  at  the  20  January  1997 
Technical  Int^change  Meeting.  Major  issues  discussed  were  Mr.  Blankenship  security 
clearance  and  the  engineering  change  proposal. 


HTSC  presented  CDRL  delivery  status,  milestones  and  short  term  goals  at  the  23 
September  1997  Technical  Interchange  Meeting. 

Appendix  A  list  the  critical  document  deliverables  in  the  order  they  were  completed. 


3.3.  Assessment 


3.3.1.  Assessment  Plan 

Planning  started  by  coordinating  a  date  with  NAIC/SCD  and  Dr 

Enoineerina  Productivity  Group.  The  assessment  workshop  was  scheduled  for  3 

September  1996  through  6  September  1996.  Mr.  Blankenship  outlined  a  workshop  p  a^^^ 

Dr  Koltun  and  Mr.  Blankenship  met  at  the  SEPG  conference  in 

and  coordinate  the  workshop.  Dr  Koltun  and  Mr.  Blankenship  completed^^^^ 

Assessment  Workshop  plan  and  discussed 

when  Mr.  Blankenship  traveled  to  Melbourne,  FL  from  24  June  Dr 

reviewed  Mr.  Blankenship's  pre-workshop  questionnaire  and  modified  the  te 
information  to  collect.  The  questionnaire  was  developed  from  the  issues  identified  in 
NAlC's  1 994  assessment. 

The  draft  plan  was  delivered  on  15  August  1996.  The  outline  of  the  workshop  plan 
follows; 

1.  INTRODUCTION 


2.  OBJECTIVES 


3.  DESCRIPTION 

3.1.  PREPARATION 

3.1.1.  QUESTIONNAIRE 
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3.1.2.  TRAINING 

3.1.3.  DOCUMENT  REVIEW 

3.2.  WORKSHOP 

3.2.1.  IN-BRIEF 

3.2.2.  WORKSHOP  ACTIVITIES 

3.2.3.  OUT-BRIEF 

3.3.  ASSESSMENT  WORKSHOP  REPORT 

4.  RESOURCES 

4.1.  TIME 

4.2.  PERSONNEL 

4.3.  OTHER  RESOURCES 

5.  RISKS 

5.1.  LOGISTICS 

5.2.  WORKSHOP  PROCESS 

5.3.  TEAM  DYNAMICS 

5.4.  ORGANIZATION 

APPENDIX  A.  PREPARATIONS  FOR  THE  ASSESSMENT  WORKSHOP 

APPENDIX  B.  PRE-WORKSHOP  QUESTIONNAIRE 

APPENDIX  C.  SCHEDULE 

APPENDIX  D.  KPA  PRIORITY  FOR  COVERAGE 

APPENDIX  E.  SUPPLIES 

APPENDIX  F.  WORKSHOP  TEAM  MEMBER  BINDERS 
APPENDIX  G.  WORKSHOP  REPORT  FORMAT 

Mr.  Blankenship  distributed  the  Pre-Workshop  Questionnaire,  Appendix  B.  of  the 
Assessment  Workshop  Plan,  to  Capt.  Fulton.  NAIC/SCD’s  Software  Process 
Improvement  Steering  Group  (SPISG)  approved  Mr.  Blankenship’s  requested  for  all  of 
SCD  to  complete  and  return  the  questionnaire  to  Mr.  Blankenship  by  12  August  1996. 


3.3.2.  Assessment  Workshop 

Mr.  Blankenship  captured  the  comments  and  responses  from  the  Pre-Workshop 
Questionnaire  (Appendix  B.  of  the  SPI  Workshop  Plan).  He  received  back  36  of  the  50 
distributed  to  SCD  personnel.  The  comments  and  responses  were  analyzed  and  used 
in  the  September  Assessment  Workshop. 
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Dr.  Phil  Kolturr,  Hams  EPG  led  .SeP^n^ber  f PI 

lonta  Kohun.  In  addhlon,  8  olher 

members  of  NAIC  attended  and  participated  in  the  workshop. 

Dr  Koltun  started  the  Assessment  Workshop  on  Tuesday  with  an  In-Brief.  The  In-bnef 
wLs  pmvide^rCol.  Wolfe  Mr.  Lush,  Mr.  Beigel,  Mr.  Mosley.  Mr.  Leaf  r^  Maj.  Burke 
and  L  workshop  Team  Members.  The  Software  Pr^esslmprover^^^^^ 

rind  th©  SoftwBr©  EnoinGGrinQ  ProcGSS  Group  (SEPQ)  p  ^ 

S  it  me  Softw”ocess  Improvement  Open  Forum,  to  rnaximrze  orfl""® 
arpocQ  tn  Dr  Koltun  At  the  open  forum  Dr.  Koltun  presented  a  briefing  on  17  ^ 

Ideas  for  Improving  How  We  Do  Software”  to  the  entire  Systems  Development  Division. 

The  actual  workshop  started  Wednesday  morning  with  the  first  key  Pfff 
Requlmments  Management  (RM).  Dr.  Koltun  started  the 

?i:n 

Cofwolfe,  L  S,  M?leigel.  Mr.  Mosley,  Mr.  Leasure,  Maj.  Burke  and 
the  Workshop  Team  Members. 

NAIC/SCD's  strengths  and  weaknesses  were  discussed  and  bf  fed  to  CoL  ^fe,  Mr 
the  workshop  findings. 


3.3.3.  Assessment  Report 

"rsSssS-SrSSHS 

Indlstaated  Ihe  resources  needed  to  work  the  items  in  the  requirement  queue.  Capt. 
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Fulton  briefed  the  SPISG  on  19  September  1996.  The  SPISG  decided  to  delay  the  SEI 
assessment,  a  CMM  Based  Appraisal  for  Internal  Process  Improvement  (CBA-IPI),  from 
March  1997  until  October  1997.  Capt.  Fulton  used  the  process  improvements 
requirements  queue  and  estimates  to  create  a  project  proposal  and  project  plan  for  the 
SPISG  and  SRB  approval. 

On  27  September  1996,  HTSC  delivered  the  draft  Assessment  Workshop  Report  from 
the  September  SPI  Workshop.  HTSC  then  worked  with  the  SEPG  to  categorize  the 
responses  to  the  4  open-ended  questions  from  the  Pre-Workshop  Questionnaire.  Mr. 
Blankenship  provided  the  SEPG  the  responses  then  sorted  them  into  the  categories 
supplied  by  the  SEPG.  Capt.  Fulton  presented  the  results  to  the  SPISG.  Col.  Wolfe, 
SC  Director,  and  Mr.  Lush,  SCD  Division  Chief  has  addressed  some  of  the  issues 
brought-up  at  the  workshop.  This  allowed  the  SPISG  to  address  any  outstanding 
issues. 

Since  the  U.S.  Air  Force  submitted  no  changes,  the  Assessment  Workshop  Plan  and 
Report  were  accepted  as  final,  on  13  December  1996. 

3.4.  A  guide  to  achieving  SEI  CMM  ievei  2  at  NAiC 

The  areas  of  concern,  as  specified  in  the  SOW  (see  Section  2.5  on  page  5),  for  study 
included  but  was  not  limited  to: 

•  Contract  Procurement  and  Management 

•  Project  Planning  and  Tracking 

•  Change  and  Status  Accounting 

•  Procedures  for  coordination  and  guidance  of  the  Software  Development  and 
Maintenance  Process 

To  investigate  the  areas  of  concern  in  the  SOW,  Mr.  Blankenship  started  with  the 
"Contract  Procurement  and  Management"  (the  CMM  calls  this  area  Software 
Subcontract  Management). 


3.4.1.  Software  Subcontracting  Management 

Mr.  Blankenship  and  Capt.  Fulton  were  identified  as  the  Software  Subcontracting 
Working  Group.  They  were  chartered  to  study  subcontract  management.  They  began 
by  creating  an  interview  script  with  questions  about  managing  contractors. 

The  next  step  was  to  interview  the  major  players  involved  in  managing  contractors.  The 
major  players  included  Mr.  Don  Quigley  (SCX),  Mr.  Dave  Leasure  (SCDD),  Mrs.  Sharon 
Cain  (SCDD),  and  Mr.  Joe  Schmalhofer  (SCD). 
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The  Subcontracting-working  group  completed  the  initial  interviews  and  delivered  the 
draft  report  on  Software  Subcontractor  Management  on  14  June  1996.  This  was  the 
first  deliverable  for  the  guide  to  achieving  CMM  level  2. 

The  next  step  was  to  review  SCO’s  Policy  and  Practices  (PnP)  documentation  on 
project  management  and  configuration  management.  Additionally,  continue  to  work 
with  chartered  working  groups  to  learn  more  about  NAIC/SCD. 


3.4.2.  Estimation  Working  Group 

Mr.  Blankenship  met  with  the  Estimation  working  group  to  define  the  scope  of  the 
working  group  and  the  requirements  to  satisfy.  The  Estimation  working  group  decided 
to  review  past  work  on  estimating  and  look  at  current  tools  and  procedures  available. 

Mr.  Blankenship  contacted  Mr.  Daniel  Ferns,  AFIT  professor,  researching  estimating 
tools  and  procedures.  Mr.  Ferns  is  a  noted  author  and  researcher  on  estimating  tools 
and  their  use.  Mr.  Blankenship  described  NAIC/SCD  and  their  project  size  and  duration 
and  Mr.  Ferns  agreed  current  tools  and  procedures  were  developed  for  much  larger 
projects.  In  order  to  use  these  tools,  NAIC/SCD  would  need  to  gather  data  to  customize 
the  tools  to  SCD’s  project  size. 

The  Estimation  working  group  defined  the  project  lifecycle  model  and  Identified  the 
points  estimation  activities  occurs.  Each  member  was  assigned  to  document  estimation 
activities  and  develop  process  descriptions.  Mr.  Blankenship  provided  guidelines  and 
suggestions  for  developing  and  documenting  their  process  descriptions. 

Emphasis  on  the  work  with  the  Estimation  Group  was  reduced  based  upon 
recommendations  made  by  the  SEPG  and  the  SPISG. 


3.4.3.  Configuration  Management  Working  Group 

The  SEPG  and  SPISG  chartered  a  Configuration  Management  working  group.  The 
Configuration  Management  Working  Group  defined  Its  scope  and  decided  to  call  itself 
the  CM  process  team  (CMpit).  The  CMpit  was  chartered  to  define  and  document  the 
CM  processes  needed  and  select  a  CM  tool  to  support  the  CM  processes.  At  the  first 
meeting  with  the  SPISG,  the  SPISG  stressed  the  need  to  acquire  a  CM  tool. 

The  CMpit  generated  a  list  of  requirements  for  a  CM  tool  and  performed  the  subsequent 
review.  General  Research  Corporation  International  personnel  gave  a  demonstration  of 
Atria’s  Clearcase  to  the  CMpit.  Atria  personnel  also  gave  a  demonstration  of  Clearcase 
and  answered  questions.  SQL  personnel  gave  a  demonstration  and  training  session  ori 
SQL’s  PCMS  CM  tool.  Sharon  Otto,  AMC/CPSS  PPSC,  sent  Mr.  Blankenship  a  copy  of 
their  CM  tool  selection  findings.  The  CMpit  briefed  the  SPISG  on  their  progress  and 
was  directed  to  acquire  a  copy  of  Atria’s  Clearcase  and  test  for  an  80  percent  solution. 
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At  the  19  September  1996  SPISG  meeting,  the  CMpit  was  directed  to  focus  on  CM 
processes  and  procedures  and  make  the  CM  tool  a  long-range  objective.  CM  went 
ahead  and  purchased  the  tool  but  will  wait  until  someone  is  trained  and  the  tool  is 
installed  on  hardware  before  changing  the  processes  and  procedures  to  use  the  tool.  A 
CM  tool  is  not  necessary  to  achieve  CMM  Level  2. 

Mr.  Blankenship  developed  and  provided  draft  copies  of  Configuration  Management 
activities  across  the  SCD  project  lifecycle  to  the  CMpit. 

Mr.  Blankenship  developed  and  provided  draft  copies  of  guidance  on  Configuration 
Control  Board  (CCB)  and  CM  audits  to  the  CMpit.  The  CM  audits  include  Physical 
Configuration  Audit  (PCA)  and  Functional  Configuration  Audit  (FCA). 

After  full  review  by  the  CMpit  and  SEPG,  Mr.  Blankenship  updated  draft  copies  of 
Configuration  Management  activities  across  the  SCD  Standard  Project  Lifecycle  to  the 
CMpit.  He  then  developed  a  new  high-level  model  of  the  SCD  Standard  Lifecycle  and 
developed  multiple  models  to  show  CM  activities  and  interfaces  to  other  processes. 


3.4.4.  Project  Planning,  Tracking,  &  Oversight 

Mr.  Blankenship  reviewed  and  provided  comments  on  SCD’s  Policy  and  Practices  (PnP) 
Chapter  5,  Project  Planning  Tracking  &  Oversight.  Mr.  Blankenship  looked  for 
weakness  as  compared  to  the  CMM  and  two  of  the  KPAs.  Suggestions  were 
documented  and  provided  to  the  SEPG. 


3.4.5.  Guide  to  CMM  Level  2 

Mr.  Blankenship  reviewed  the  findings  from  the  1 994  SPA  and  documented  changes  in 
the  report. 

Mr.  Blankenship  delivered  the  draft  of  “A  Guide  to  Achieving  SEI  CMM  Level  2  at  NAIC” 
(CDRL  A004)  on  15  August  1996.  This  guide  covered  three  areas:  Subcontract 
Management,  Project  Planning  and  Tracking,  and  Configuration  Management. 

Mr.  Blankenship  delivered  the  Final  “A  Guide  to  Achieving  SEI  CMM  Level  2  at  NAIC” 
technical  information  report  (CDRL  A004)  on  15  October  1996.  This  report  is 
summarized  in  Appendix  C. 


3.5.  Presentation  or  Background  Material 
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3.5.1.  Requirements  &  Design 


Mr.  Blankenship  conducted  two  lead  developers  interview  sessions  on  how  NAIC 
manages  requirements.  Then  worked  with  Capt.  Fulton  to  engineer  changes  to  the 
project  initiation  to  include  more  discipline  on  requirement  management.  Capt.  Fulton 
provided  examples  of  documented  requirements. 

Mr.  Blankenship  developed  some  guides  to  use  for  modeling  requirements.  This 
included  an  introduction  to  structural  analysis  and  provided  some  NAIC  standards  for 
data  flow  diagrams  (DFDs),  entity  relationship  diagrams  (ERDs),  and  state  transition 
diagrams  (STDs).  Mr.  Blankenship  also  provided  an  introduction  to  Concept  Maps 
(CMAPS)  for  object  oriented  problem  analysis.  Additionally,  some  sample  problems 
with  DFD,  ERD,  STD,  and  CMAPS  were  included.  These  guides  were  provided  in  draft 
form  to  provide  SCD  analyst  a  starting  point.  SCD  analyst  should  modify  and  include 
the  object-oriented  method  of  choice  and  any  other  standard  analysis  or  modeling 
methodology. 


3.5.2.  SEI 

HTSC  worked  with  other  NAIC/SCD  contractors  to  supply  a  coordinated  effort. 

Capt.  Coquillard,  Capt.  Fulton,  Capt.  Carlin  (all  of  NAIC/SCDQ),  Mr.  Blankenship 
(HTSC),  and  Mr.  Lynn  Carter  (SEI)  met  to  plan  future  efforts  for  NAIC/SCD’s  process 
improvements. 

Mr.  Blankenship  met  with  Dr.  Lynn  Carter  (SEI),  Mr.  Dan  Green  (SEI),  Capt.  Ed 
Coquillard,  and  Capt.  Greg  Fulton.  SEI  planned  to  work  with  NAIC/SCD  and  HTSC  to 
define  management  processes  to  prioritize  work  requests  and  manage  a  set  of  projects. 
The  most  important  would  be  a  System  Resources  Board  chapter. 


3.5.3.  SRB  Chapter 

Mr.  Blankenship  worked  with  Capt.  Greg  Fulton  and  Capt.  Ed  Coquillard  to  create  the 
first  draft  of  a  working  paper  on  the  SRB.  SEI  used  the  working  paper  and  worked  with 
NAIC/SCD  and  HTSC  to  define  management  processes  to  prioritize  work  requests  and 
manage  a  set  of  projects. 


3.5.4.  Process  Briefing 

Mr.  Blankenship  worked  with  Capt.  Greg  Fulton  and  Capt.  Ed  Coquillard  to  create,  and 
prepare  a  briefing  on  NAIC/SCD’s  process  architecture  for  the  1 997  HQ  AFCA  Software 
Process  Improvement  (SPI)  Workshop. 
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This  briefing  was  a  great  beginning  for  an  introduction  to  the  NAIC/SCD  process.  It 
provides  background  and  future  of  the  process. 


3.5.5.  Training  Program 


Mr.  Blankenship  created  a  high-level  template  for  process  overview  training.  This 
training  has  three  levels.  1)  A  single  page  showing  the  process  ^ 

phases,  milestones,  and  major  review  points.  2)  A  page  per  phase  with  the  key  things 
that  occur  in  the  phase  with  inputs,  outputs,  and  high-level  activities.  3)  ^ne  or  more 
pages  containing  details  on  the  activities  for  a  given  phase  identifying  the  individual 
responsible  to  complete  the  activity. 

He  then  participated  in  developing  and  delivering  process  overview  training  for  version 
3.1  of  the  Application  Development  Process.  In  addition,  Mr.  Blankenship  and  Capt. 
Fulton  developed  and  delivered  training  on  risk  management,  requirements 
management,  and  software  estimation. 


Mr.  Blankenship  developed  orientation-training  outlines  for  Software  Quality  Assurance, 
Configuration  Management,  and  System  Testing.  This  standard  template  provides 
guidance  to  the  functional  areas  when  developing  their  own  process  training. 

ompleted  reviews  and  revisions  of  orientation  training  slides  for  three  of  the  software 
service  branches.  This  orientation  training  was  for  version  3.2  of  NAIC/SCD  Policy  and 
Practices  (PnP).  This  provided  orientation  training  on  Software  Quality  Assurance, 
Software  Configuration  Management,  and  System  Testing. 


Mr.  Blankenship  created  a  survey  to  collect  information  on  training  received  by  SCD 
personnel.  This  was  a  beginning  for  creating  a  SCD  training  program. 


3.5.6.  Risk  Management 

Mr.  Blankenship  developed  and  delivered  a  briefing  to  introduce  risk  management.  Col. 
Wolfe  requested  additional  information.  Mr.  Blankenship  researched  prior  issues  ot 
STSC’s  Crosstalk  and  found  the  information  for  Col.  Wolfe. 


3.5.7.  Software  Life-Cycle  Model 

Mr.  Blankenship  developed  and  provided  background  information  and  an  introduction  to 
software  life-cycle  process  models  with  the  advantages  and  disadvantages. 

NAIC/SCD  tends  to  think  in  terms  of  a  project  life  cycle  and  not  a  systeni.  Mr. 
Blankenship  suggested  SCD  investigate  a  system  software  life-cycle  process  model. 
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3.5.8.  Metrics 


Mr.  Blankenship  developed  an  outline  for  metrics  spreadsheet  for  project  data.  Each 
System  has  a  worksheet  with  multiple  project  tables  containing  key  project  information 
for  each  phase  of  the  lifecycle.  Then  a  summary  worksheet  for  all  systems  sorted  Into 
tables  for  small,  medium,  and  large  systems.  The  key  project  information  includes  effort 
(hrs),  duration  (days),  document  defects,  product  defects,  document  quantity  (pages), 
product  quantity  (LOG),  product  productivity  (LOC/hr),  document  productivity  (pages/hr), 
and  defect  rate  (defects/LOC  &  defects/pages).  From  this  outline,  Capt.  Fulton 
developed  a  working  prototype  to  demonstrate  the  concept  to  project  managers. 

Mr.  Blankenship  developed  a  metrics  spreadsheet  with  pivot  tables  on  time  accounting 
data.  These  tables  allow  managers  to  evaluate  individuals  or  projects  across  time 
periods  looking  for  trends.  One  pivot  table  pivots  on  projects  and  shows  time  charged 
by  person  for  each  month  from  April  1997  through  November  1997.  The  second  pivot 
table  pivots  on  people’s  names  and  shows  the  time  charged  to  project  codes  for  each 
month  from  April  1 997  through  November  1 997. 

Mr.  Blankenship  developed  an  additional  metrics  spreadsheet  with  pivot  tables  from 
time  accounting  data.  These  tables  have  trend  information  on  cost  of  projects  by 
phase.  This  information  will  be  used  by  future  projects  to  improve  phase  estimation. 
This  was  the  first  step  to  gather  the  information  to  populate  the  project  metrics 
spreadsheet. 

Mr.  Blankenship  developed  an  ’awk’  language  script  to  clean  up  and  standardize  the 
data  from  the  time  accounting  database  for  the  information  by  phase  pivot  tables.  He 
demonstrated  good  programming  practices  by  using  a  program  header  complete  with 
usage  information  and  change  history. 


3.5.9.  Application  Classes  &  Artifacts 

Mr.  Blankenship  started  collecting  information  to  develop  application  classes  to 
determine  standard  artifacts.  The  standard  artifacts  were  to  be  used  In  program 
management  to  plan  code  and  unit  test  phase  to  address  project  planning  and  tracking 
weaknesses  found  in  the  assessment.  After  inten/iewing  multiple  people,  Mr. 
Blankenship  canceled  the  effort.  Lack  of  robust  designs  limited  developers 
understanding  and  need  for  a  defined  set  of  standard  artifacts. 


3.6.  A  final  scientific  and  technical  report  (CDRL  A005) 

The  method  used  to  develop  this  report  was  to  gather  all  pertinent  Information  and 
document  the  major  efforts  at  NAIC.  Then  document  the  lessons  learned  and  any 
recommendations  for  the  future. 
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A  NAIC/SCD  PnP  Configuration  Management  Chapter 


3.7. 

S“n.s^XradLes  ^  occur'and  the  handbook  documents  how  to 
perform  the  activities. 

solution  then  participated  in  a  training  session. 

He  delivered  the  draft  Configuration  Managemeht  chap^^ 

NAIC/SCD  PnP  Configuration  Management  Chapter  was  deli 


3.9.  Additional  Support 

This  section  includes  other  eHorts  that  were  not  directly  tied  to  deliverables. 

3.9.1.  PnP  Review 

Plan. 

Mr.  Blankenship  and  Capt.  Fulton  conducted  an  in-hous^d^^^^^  oUhe  NAIC/SCD 

development  processes  with  CMM  The  Audit  provided  SCD 

was  to  ensure  existing  processes  fully  “mPlV to  the 
with  improvements  and  long  term  suggestions.  These  suggestions 
requirement  list  lor  the  next  version  of  the  process. 


3.9.2.  Final  Push  Project 

The  SEPG  and  SPISG  chartered  Cap^ 

address  the  weakness  found  in  the  Septem  q«h  started  to  work  on  the 

the  Formal  Project  KickKtff  for  Me^eloplltg  gS^^ 

efforts  of  a  project. 
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3.9.3.  SQA 


Mr.  Blankenship  modified  SQA’s  problem  resolution  process.  The  modified  process 
allows  individuals  to  solve  problems  at  the  lowest  levels. 

He  then  completed  formatting  the  Software  Quality  Assurance  chapter  with  version  3.2 
style  changes  to  identify  policy  and  process  artifacts.  Additionally,  he  rewrote  the 
quality  assurance  reporting  process  and  redesigned  the  compliance  resolution  process. 


3.9.4.  Technology  Adoption  Plan 

Mr.  Blankenship  developed  a  plan  to  implement  v3.2  of  NAIC/SCD  Policy  and  Practices 
(PnP)  to  utilize  the  technology  adoption  curve  and  pilot  projects  to  work  out  problems 
and  improve  the  process-training  suite.  He  presented  a  briefing  to  the  SRB  on 
technology  adoption  and  the  new  plan. 


3.9.5.  CBA-IPI  Assessment 

To  prepare  for  the  CMM  Based  Appraisal  for  Internal  Process  Improvement  (CBA-IPI), 
Mr.  Blankenship  developed  a  project  information  table  of  current  and  past  BCD  projects 
for  assessment  preparation.  The  information  was  used  to  select  representative  projects 
for  the  assessment.  He  validated  and  verified  the  information  against  SQA  records.  He 
delivered  the  project  information  table  to  Mr.  Joe  Beigel,  SCDS,  to  modify  and  maintain 
for  a  project  status  table.  Additionally,  he  reviewed  cross-reference  between  the 
Capability  Maturity  Model  (CMM)  and  the  PnP,  chapters  and  artifacts,  for  version  3.1  of 
NAIC/SCD  development  process. 

Mr.  Blankenship  then  proctored  the  software  process  maturity  questionnaire  for  the 
project  leaders  selected  for  the  software  process  assessment. 

Mr.  Blankenship  concluded  preparations  for  the  software  process  assessment  by 
reviewing  organizational  information  needed  for  the  software  process  assessment  and 
preparing  process  binders. 

He  then  attended  the  assessment  team  member  training. 

Mr.  Blankenship  participated  as  a  coach  and  team  member  of  the  CBA-IPI.  The 
software  process  assessment  compared  NAIC/SCD’s  policies  and  practices  against  the 
Capability  Maturity  Model  from  Software  Engineering  Institute  at  Carnegie  Mellon 
University. 
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Mr  RIankenshiD  developed  and  provided  input  for  the  software  process  jssessr^nt 
final  report.  In  addition,  Mr.  Blankenship  developed  further  information  on  the  comm 
thread  between  weaknesses  and  issues  found  in  the  assessment. 


Mr.  Blankenship  developed  a  set  of  tactical  activities  to  address  the  weaknesses  and 
issues  found  in  the  assessment. 


20 


4.  Recommendations 


There  were  three  main  issues  found  in  the  CBA-IPI: 

•  Granularity  of  activities  supplied  by  project  plans  (limited  insight  of  actual  progress 
by  development  teams) 

•  Lack  of  product  standards  and  evaluations 

•  Limited  evidence  of  configuration  management 

NAIC/SCD  was  able  to  satisfy  the  first  two  by  adopting  the  personal  software  process 
(PSP).  Two  trained  instructors  should  be  able  to  teach  PSP  to  the  remaining  personnel. 
With  the  increased  discipline  from  PSP,  developers  will  have  better  planning  and 
estimating  skills.  Additionally,  PSP  teaches  peer  review  techniques  that  will  help  with 
the  product  standards  and  evaluations.  SCO  must  enforce  current  PnP  on  configuration 
management:  create  and  maintain  baselines;  and  place  all  systems  under  configuration 
control. 

In  the  final  days  of  this  effort  key  people  involved  with  software  process  improvement 
retired  or  separated  from  the  Air  Force.  A  software  process  improvement  program  is  a 
long-term  commitment.  In  order  to  sustain  a  long-term  effort.  Air  Force  units  need  to 
find  a  way  to  recruit  and  retain  trained  software  process  improvement  engineers  and 
qualified  software  engineers.  In  order  to  sustain  a  long-term  effort  and  commitment,  the 
reward  and  punishment  system  must  have  clear  goals  and  reward  those  individuals  that 
support  and  advance  software  process  improvement. 
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5.  Lessons  Learned 


5.1.  Security  Clearance 

The  time  it  took  to  investigate  and  grant  his  clearance  J  ? 

limited  Mr.  Blankenship’s  access  to  NAIC/SCD  personnel.  This  ^  . 

needed  to  learn  the  people  and  culture  and  limited  the  return  on  investment  for 

improvements. 

5.2.  Individual  Vs  Team  Approach  to  Process  Improvement 

NAIC/SCD’s  principal  individual  involved  with  process  improvement  was  the  Chair  of  the 
SEPG  While  an  individual  can  make  progress  it  usually  takes  longer  to  get  eyerybo  y 
else  involved.  If  teams  are  used  then  the  team  members  have  buy-in  to  the  team 
suggestions  and  more  people  are  involved  in  spreading  the  word  to  non-team  members. 

5.3.  “As-is"  Process  or  "To-be"  Process 

NAIC/SCD  had  some  of  the  typical  problems  associated  with  software  P^cess 
improvements.  Total  Quality  Management  theory  teaches  to  document  the  cu^t 
oractices  (As-is)  then  look  for  improvements,  document  the  future  practices  (To-be)  and 
the  Son.  SCD  always  documented  the  To-be"  state  and  did  not  always 

define  the  "As-is"  or  the  transition. 


5.4.  Most  Difficult  Problems 

The  most  difficult  problems  were  the  culture  barriers  to  change.  While 

was  stressing  the  need  to  improve,  the  people  did  not  hear  or  agree  with  the  need  and 

direction  of  the  government. 
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Appendix  A:  Contract  Deliverables 

This  appendix  lists  the  final  critical  contract  deliverable  documents  developed  by  Harris 

Technical  Service  Corporation.  Documents  are  listed  in  chronological  order. 

1.  CDRL  A004,  A  Guide  to  Achieving  SEl  CMM  Level  2  at  NAIC,  Harris  Technical 
Services  Corporation  Document  Reference  #  1070-011,  Fairview  Heights,  IL.  15 
October  1996 

2.  CDRL  A002,  Assessment  Workshop  Plan  and  Report  ,  Harris  Technical  Services 
Corporation  Document  Reference  #  1070-014,  Fairview  Heights,  IL.  13  December 
1996 

3.  CDRL  A007,  NAIC/SCD  PnP  Configuration  Management  Chapter,  Harris  Technical 
Services  Corporation  Document  Reference  #  1070-025,  Fairview  Heights,  IL.  23 
October  1997 

4.  CDRL  A005,  Final  Scientific  and  Technical  Report. 
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Appendix  B.  Workshop  Findings 


Thisappentf«oon,ains.heMngs^^^^^^ 

eLSpA^"-mrMowfng  Jctions  contain  the  actual  findings  with  the  strengths  and  weaknesses. 


B.1 .  KPA  Rating  Summary 


crtft\A/ar<a  Ri ihmntract  ManaaGiTient 


Integrated  Software  Management 
Legend 

NS  -  Not  Satisfied 
PS  -  Partially  Satisfied 
S  -  Satisfied _ 


B.2.  GENERAL  OBSERVATIONS 


Environmentat  factors  that  drive  software  organizations  to  greater  process  discipline  and  to  greater 
product  standardization  are  lacking  for  the  most  part  at  NAIC. 


multi-version  software  installed  at  geographically  dispersed  sites 
interoperability  with  external  software  systems 

ha?d  response  time  or  throughput  requirements  or  constraints  on  memory,  bandwidth,  etc. 
customers  with  fixed  and  demanding  schedules 
sharing  of  staff  across  application  domains  and  projects 
pressure  to  accumulate  reusable  component  assets 
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•  requirement  to  support  customers  with  older  versions  of  software 

B.2.1.  STRENGTHS 

New  process  elements  have  been  defined  and  documented 

B.2.2.  WEAKNESSES 

Adoption  of  the  new  process  by  the  projects  is  still  in  process 

B.3.  REQUIREMENTS  MANAGEMENT  (PS) 

B.3.1.  STRENGTHS 

Change  requests  are  entered  in  CARMS,  analyzed  according  to  defined  criteria,  and  commissioned  by 
the  SRB  and/or  domain  managers 

System  requirements  are  documented  in  the  Consolidated  Systems  Document 
Appropriate  functions  are  involved  in  review  of  project  requirements 

B.3.2.  WEAKNESSES 

Some  confusion  in  terminology  exists  in  distinguishing  software  requirements  from  statements  of  need 
Hard  requirements  may  not  always  be  separated  out  from  customer  wish  lists 
A  perception  exists  that  “requirements  creep”  still  occurs 

Procedures  and  tools  have  not  been  established  to  ensure  that  tractability  occurs  from  requirements  to 
design  to  code  to  test 

B.4.  SOFTWARE  PROJECT  PLANNING  (PS) 

B.4.1.  STRENGTHS 

A  standard  is  in  place  for  SDP  content 

A  standard  basis  of  estimate  is  in  place  for  estimating  QA  and  CM  effort 
Developers  participate  in  the  commitment  process 
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Use  of  CARMS  enables  management  to  gauge  more 
services 


accurately  the  demand  for  software  development 


B.4.2.  WEAKNESSES 

Little  historical  data  is  available  from  completed  projects  to  use  in  estimating  new  tasks 
A  checklist  is  lacking  to  cover  all  items  that  must  be  covered  in  the  estimation  process 
Detailed  milestone  charts  are  not  consistently  used  to  represent  project  schedules 
Few  perceived  consequences  exist  for  late  completion  of  projects 

Current  resource  commitments  not  always  considered  when  negotiating  resources  for  a  new  project 


B.4.  SOFTWARE  PROJECT  TRACKING  &  OVERSIGHT  (PS) 

B.4.1.  STRENGTHS 

PMRs  and  weekly  status  reports  are  used  to  track  progress  against  plan 
Standard  tools  are  used  for  planning  and  tracking 

Actual  resource  expenditure  is  tracked  to  standard,  pre-defined  lifecycle  phases 
B.4.2.  WEAKNESSES 

Corrective  actions  not  always  appropriately  taken  when  deviations  from  plan  occur 
Limited  corporate  view  of  resources  available  to  augment  projects  that  need  additional  help 
Experience  in  use  of  AutoPLAN/AutoTEAM  is  limited 

B.5.  SOFTWARE  QUALITY  ASSURANCE  (S) 

B.5.1.  STRENGTHS 
A  standard  SQA  plan  has  been  developed 

Good  checklists  and  agendas  are  used  for  SQA's  reviews  of  work  products 
SQA  staffing  level  appears  adequate 
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SQA  has  been  effective  at  reporting  and  heiping  resolve  noncompliance  issues 


B.5.2.  WEAKNESSES 

None  Noted 

B.6.  SOFTWARE  CONFIGURATION  MANAGEMENT  (PS) 

B.6.1.  STRENGTHS 

Developmental  CM  has  proven  adequate  for  most  project  circumstances  to  date 

Most  work  products,  including  plans  and  documentation  are  placed  under  configuration  control 

Procedures  for  analysis  and  review  of  proposed  changes  are  in  place 

Steps  are  taken  to  verify  that  approved  changes  have  been  incorporated  into  the  code  and  have  been 
documented  in  a  configuration  file 

B.6.2.  WEAKNESSES 

Changes  to  systems  software  periodically  undermines  the  integrity  of  operational  applications  software 

Development  tools/scripts/utilities  and  COTS  packages  are  not  formally  controlled,  nor  are  test  data  or 
test  scripts 

In  most  cases,  production  code  is  not  under  CM  control 

A  consensus  has  not  been  solidified  on  procedures  for  transitioning  software  from  the  developer  to  the 
user  and  to  configuration  control 

Systematic  recording  of  program  trouble  reports  does  not  appear  to  be  occurring  routinely 

B.7.  ORGANIZATION  PROCESS  FOCUS  (PS) 

B.7.1.  STRENGTHS 

Responsibility  for  initiating  and  coordinating  process  improvement  activities  has  been  vested  in  an  SEPG 
Involvement  in  SPI  activities  is  widespread 

SPI  efforts  are  planned  and  managed  as  thoroughly  as  the  organization’s  development  projects 
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B.7.2.  WEAKNESSES 


Software  process  training  is  incomplete 

B.8.  ORGANIZATION  PROCESS  DEFINITION  (PS) 

B.8.1.  STRENGTHS 

Procedures  have  been  defined  to  cover  the  way  that  new  policies,  procedures,  and  guidebooks  are 
developed 

The  process  asset  library  has  been  established  and  populated  with  model  work  products,  templates, 
checklists,  and  works  in  progress;  process  assets  are  available  online 

Proposed  changes  to  the  command  media  are  handled  expeditiously 

B.8.2.  WEAKNESSES 

Some  policies  and  procedures  are  still  in  development 
Limited  guidelines  for  tailoring  standard  processes 

Project  size  and  effort  actuals  are  not  recorded  in  the  PAL  to  aid  future  estimation  activities 

Lessons  learned  from  prior  projects  are  not  always  captured  and  accessible  to  all  that  might  benefit  from 
them 

B.9.  SOFTWARE  PRODUCT  ENGINEERING  (NS) 

B.9.1.  STRENGTHS 

Some  standardization  of  tools  and  implementation  approach  occurs  in  the  database  area 
A  library  of  code  samples  is  in  development  to  encourage  learning  and  reuse 

B.9.2.  WEAKNESSES 

Standard  ways  of  performing  analysis  and  design  have  not  yet  been  identified 
A  large  number  of  tools  and  languages  are  used  for  implementation 
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Standards  do  not  yet  exist  for  a  number  of  software  work  products 

B.10.  TRAINING  PROGRAM  (NS) 

B.10.1.  STRENGTHS 

Some  training  needs  are  identified  as  part  of  civilian’s  annual  performance  appraisal 
Position  descriptions  exist 

B.10.2.  WEAKNESSES 

Training  needs  are  not  systematically  identified  for  SCD  personnel 
A  training  program  has  not  yet  been  established 
Priorities  for  training  have  not  been  clarified 
Effectiveness  of  delivered  training  is  not  routinely  assessed 

B.11.  PEER  REVIEWS  (PS) 

B.11.1.  STRENGTHS 

Standards  exist  for  the  content  and  conduct  of  major  reviews,  and  for  the  follow-up  activities  afterward 

B.11.2.  WEAKNESSES 

Organizational  manpower  planning  does  not  consistently  account  for  staff  time  to  support  reviews 

Code  reviews  are  not  required 

A  peer  review  handbook  does  not  yet  exist 

Training  in  review  technique  has  not  been  formally  delivered 
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Appendix  C.  Summary  of  "A  Guide  to  achieving  SEI  CMM  Level  2  at 
NAIC" 

This  appendix  contains  the  executive  summary  and  suggestions  from  the  report. 


C.1.  Executive  Summary 

The  CMM’s  Level  2  KPA:  Software  Subcontract  Management  does  not  apply  to  SCD.  There  are 
some  informal  SCD  roles  to  document  in  SCD’s  Policy  and  Practices  Manual.  Collection  and 
use  of  historical  data,  documented  correction  actions,  periodic  project  management  reviews,  and 
risk  management  practices  will  greatly  help  SCD  toward  satisfying  the  goals  for  project  planning 
and  tracking.  There  is  still  limited  CM  discipline  in  SCD.  Policies  and  practices  need  to  be 
defined  and  enforced. 


C.2.  Report  Suggestions 

C.2.1.  Subcontractor  Management 

Rome  Laboratory  and  NAIC/SCX  are  the  process  owners  for  the  selection  and  management  of 
software  subcontractors,  SCD  personnel  have  roles  in  these  efforts.  SCD’s  Policy  and  Practices 
Manual  (PnP)  Chapter  12  must  document  these  roles  and  concentrate  on  things  SCD  personnel 
need  to  know  about  subcontracting  and  /  or  who  to  contact  about  working  with  subcontractors. 
Roles  identified  to  date  include: 

•  Working  with  Intel  customers  to  define  needs  and  requirements 

•  Review  Initial  Estimates 

•  Review  &  Advise  on  PDL 

•  Review  &  Advise  on  Execution  Plan 

•  Attend  Monthly  Reviews 

•  Test  Delivered  System 

Above  all  else  PnP  Chapter  12  needs  to  clearly  state  and  explains  why  SSM  does  not  apply  to 
NAIC.  Assessors  want  to  know  and  see  it  documented  that  an  informed  decision  has  been  made. 


C.2.2.  Project  Planning,  Tracking  and  Oversight 

PnP  Chapter  5,  Project  Planning,  Tracking  and  Oversight  incorporate  some  project  planning 
suggestions.  Suggestions  included: 

•  Clearly  state  policy 
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•  Change  "should  or  may"  verbiage  to  say  will  (What  is  required) 

•  Or  under  what  set  of  circumstances  can  this  procedure  be  eliminated 

•  And  who  can  approve  the  elimination. 

•  Change  approval  for  all  work  to  the  SRB  or  resource  holders  (some  small  task  could  get 
approval  for  work  and  scheduled  without  the  SRB  or  resource  holder  approval). 

•  Add  a  conunitment  procedure  for  all  affected  groups  to  agree  to  project  commitments. 

Concerns  for  Tracking  and  Oversight  include; 

•  Tracking  actual  results  and  performance  against  the  project  plan. 

•  Corrective  action  for  results  and  performance  deviations  from  the  project  plan 

•  Standards  (templates)  for  periodic  program  management  reviews  (PMRs) 

•  Risk  management 

Some  of  the  concerns  will  work  out  as  Chapter  5  is  used  by  more  of  SCD  and  practitioners  gain 
more  knowledge  and  improve  estimating  procedures. 


C.2.3.  Configuration  Management 

The  April  1994,  Software  Process  Assessment  (SPA)  Findings  Report  said: 

•  Software  Configuration  Management  is  not  widely  implemented.  This  finding  is  evidenced 
by: 

•  Resources  and  tools  for  CM  are  limited  or  unavailable. 

•  Procedures  are  seldom  formalized. 

•  Existing  procedures  are  usually  not  enforced. 

•  Implementation  is  often  left  up  to  the  individual. 

•  Requirements,  design  and  documentation  are  rarely  baselined  or  controlled. 

The  following  improvements  have  been  made  since  the  April  1994  SPA: 

•  User’s  requirements  are  loaded  into  ARS  Carms. 

•  Domain  managers  monitor  and  manage  requirements  for  their  domains. 

•  The  SRB  informally  performs  some  of  the  CCB  tasks 

•  AutoPLAN  can  manage  the  schedule  and  changes  to  the  schedules  throughout  a  project’s 
lifecycle.  AutoPLAN  allows  multiple  versions  and  baselines  for  a  project. 

•  A  draft  PnP  Chapter  1 1,  Configuration  Management  has  been  written. 

•  A  CM  Working  Group  is  working  to  recommend  a  commercial-off-the  shelf  (COTS)  CM 
tool.  Tool  requirements  were  defined  before  tool  evaluations. 

•  Policy  statements  for  CM  to  take  charge  and  move  all  project  material  to  archival  storage 
after  each  project. 
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While  this  may  seem  like  a  lot  of  improvements,  there  are 

Without  addressing  and  solving  CM  issues  no  quality  improvement  effort  can  succeed.  U 
risks  noted  include: 

•  CM  procedures  are  limited  and  not  enforced. 

•  Developers  control  and  manage  production  systems. 

•  Configurable  items  are  not  always  controlled  by  NAIC. 

•  Configurable  items  are  not  viewed  as  NAIC  asset. 

•  Configurable  items  are  stored  in  directories  owned  by  the  developers. 

•  Developers  think  and  act  like  they  "own"  the  source  they  develop. 

•  Developers  install  and  manage  production  systems. 

•  Developers  developing  on  production  systems. 


C.2.4.  Change  and  Status  Accounting 

At  this  time  with  limited  baselined  systems,  change  and  status  accounting  is 

ntc  in  ARS  Carms  system  Domain  managers  should  penodically  report  on  the 
XTm  ui«  ta  could  contain  the  number  of  new  requimmen^. 

old  requirements  (not  yet  scheduled),  iu-progress  old 

J^reMmXpM"  old  r^uteLuB  tL  old  requirement  and  add  a  new  one), 

or  requirements  satisfied  and  system  installed  on  production  machine. 

Work  nroducts  documentation,  and  deliverables  need  to  be  identified  as  Configurable  Iteim 

C  nr^e^es  need  to  s  ate  where  CIs  are  created  and  when  they  are  placed  under  CM 
(CIS).  The  precedes  neenm  ^our. 

Xu1do„"5;s.™  wm  bTcrerdlnd  their  release  controlled  by  CM  personnel.  Release 
procedures  will  need  to  be  defined. 

pSom  red  role.  Additionally,  CM  will  periodically  report  on  any  changes  to  the  CM 
repository. 
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Appendix  D.  Acronyms,  Offices,  and  Organizations 
D.1.  Acronyms 

CM  Configuration  Management 

CMM  Capability  Maturity  Model 

SEPG  Software  Engineering  Process  Group 

SPISG  Software  Process  Improvement  Steering  Group 

KPA  Key  Process  Area 

RM  Requirements  Management 

PnP  Policy  and  Practices 

CMpit  Configuration  Management  Process  Team 

IPI  Internal  Process  Improvement 

PSP  Personal  Software  Process 

D.2.  Offices  and  Organizations 

Harris/HTSC 

Harris/EPG 

NAIC/SCX 

NAIC/SCDD 

NAIC/SCD 

Rome  Laboratory  (now  AFRL/IF) 

AFIT  Air  Force  Institute  of  Technology 

SEI  Software  Engineering  Institute 
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MISSION 

OF 

AFRL/INFORMATION DIRECTORATE  (IF) 


The  advancement  and  application  of  Information  Systems  Science 
and  Technology  to  meet  Air  Force  unique  requirements  for 
Information  Dominance  and  its  transition  to  aerospace  systems  to 
meet  Air  Force  needs. 


